{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![model](img/encoder-decoder.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "此分类模型是来自序列模型[Convolutional Sequence to Sequence Learning](https://arxiv.org/pdf/1705.03122.pdf)，整体构架如上图所示。\n",
    "\n",
    "原论文是用来做语言翻译，这里我将稍微修改用来做问答中的slot filling和intent detection联合建模。\n",
    "\n",
    "本项目中的图片和原始代码是改自https://github.com/bentrevett/pytorch-seq2seq 在此非常感谢作者实现了这么通俗易懂的代码架构，可以让其它人在上面进行修改。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![model](img/encoder.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Encoder:\n",
    "    \n",
    "    1.句子token和其对应的position经过embedding后，逐元素加和作为source embedding。\n",
    "    \n",
    "    2.source embedding经过： 线性层 -> 卷积块后得到的特征 -> 线性层。\n",
    "    \n",
    "    3.以上的输出和source embedding进行残差连接。\n",
    "    \n",
    "    4.以上的输出，我这里加了一个平均池化后进入线性层，预测输出intent概率。（这时是用来做intent detection，即意图识别）\n",
    "    \n",
    "    5.原模型的encoder的输出包含两部分，一个是卷积输出；一个是卷积输出 + source embedding -> 这两个输出将用于deocder中的卷积块中计算相应attention context。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![model](img/encoder-conv.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Encoder-conv(encoder中的卷积块):\n",
    "    \n",
    "    1.卷积块的初始输入是 source embedding加一个线性层，padding后输入卷积。\n",
    "    \n",
    "    2.卷积后经过glu激活函数\n",
    "    \n",
    "    3.激活后的输出和padding后的输入进行残差连接，进入下一个卷积块。\n",
    "    \n",
    "    4.最终输出卷积特征。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![model](img/decoder.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Decoder:\n",
    "    \n",
    "    1.target标签的token和其对应的position经过embedding后，逐元素加和作为target embedding。\n",
    "    \n",
    "    2.target embedding经过线性层的输出和target embedding -> 卷积块后得到的特征 -> 线性层。\n",
    "    \n",
    "    3.再一次经过线性层输出预测slot标签概率。\n",
    "    \n",
    "    注：可以到deocder的卷积块的输入还包含还来encoder的两个输出conved,combined"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![model](img/decoder-conv.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Decoder-conv(decoder中的卷积块):\n",
    "    \n",
    "    1.卷积块的初始输入包含4个部分，分别是：target embedding; 经过一个线性层的target embedding; encoder的卷积块输出conved； encoder联合了source embedding和卷积块的输出conved的联合输出combined。\n",
    "    \n",
    "    2.与encoder的卷积块类似，卷积后经过glu激活函数\n",
    "    \n",
    "    3.激活后的输出和target embedding; encoder conved; encoder combined一起计算attention。\n",
    "    \n",
    "    4.经过以上计算的输出，和padding后的输入进行残差连接。\n",
    "    \n",
    "    5.以上的输出进入下一个卷积块。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from torchtext import data, datasets\n",
    "import pandas as pd\n",
    "import pickle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "base_dir = os.path.abspath(os.path.join(os.getcwd(), \"..\"))\n",
    "atis_data = os.path.join(base_dir, 'atis')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "'''\n",
    "build train and val dataset\n",
    "'''\n",
    "    \n",
    "tokenize = lambda s:s.split()\n",
    "\n",
    "SOURCE = data.Field(sequential=True, tokenize=tokenize,\n",
    "                    lower=True, use_vocab=True,\n",
    "                    init_token='<sos>', eos_token='<eos>',\n",
    "                    pad_token='<pad>', unk_token='<unk>',\n",
    "                    batch_first=True, fix_length=50,\n",
    "                    include_lengths=True) #include_lengths=True为方便之后使用torch的pack_padded_sequence\n",
    "\n",
    "TARGET = data.Field(sequential=True, tokenize=tokenize,\n",
    "                    lower=True, use_vocab=True,\n",
    "                    init_token='<sos>', eos_token='<eos>',\n",
    "                    pad_token='<pad>', unk_token='<unk>',\n",
    "                    batch_first=True, fix_length=50,\n",
    "                    include_lengths=True) #include_lengths=True为方便之后使用torch的pack_padded_sequence\n",
    "LABEL = data.Field(\n",
    "                sequential=False,\n",
    "                use_vocab=True)\n",
    "\n",
    "train, val = data.TabularDataset.splits(\n",
    "                                        path=atis_data,\n",
    "                                        skip_header=True,\n",
    "                                        train='atis.train.csv',\n",
    "                                        validation='atis.test.csv',\n",
    "                                        format='csv',\n",
    "                                        fields=[('index', None), ('intent', LABEL), ('source', SOURCE), ('target', TARGET)])\n",
    "\n",
    "SOURCE.build_vocab(train, val)\n",
    "TARGET.build_vocab(train, val)\n",
    "LABEL.build_vocab(train, val)\n",
    "\n",
    "train_iter, val_iter = data.Iterator.splits(\n",
    "                                            (train, val),\n",
    "                                            batch_sizes=(64, len(val)), # 训练集设置为32,验证集整个集合用于测试\n",
    "                                            shuffle=True,\n",
    "                                            sort_within_batch=True, #为true则一个batch内的数据会按sort_key规则降序排序\n",
    "                                            sort_key=lambda x: len(x.source)) #这里按src的长度降序排序，主要是为后面pack,pad操作)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save source words\n",
    "source_words_path = os.path.join(os.getcwd(), 'source_words.pkl')\n",
    "with open(source_words_path, 'wb') as f_source_words:\n",
    "    pickle.dump(SOURCE.vocab, f_source_words)\n",
    "\n",
    "# save target words\n",
    "target_words_path = os.path.join(os.getcwd(), 'target_words.pkl')\n",
    "with open(target_words_path, 'wb') as f_target_words:\n",
    "    pickle.dump(TARGET.vocab, f_target_words)\n",
    "    \n",
    "# save label words\n",
    "label_words_path = os.path.join(os.getcwd(), 'label_words.pkl')\n",
    "with open(label_words_path, 'wb') as f_label_words:\n",
    "    pickle.dump(LABEL.vocab, f_label_words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torch.optim as optim\n",
    "from torch.optim import lr_scheduler\n",
    "import math\n",
    "from apex import amp\n",
    "import time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
    "SEED = 1234\n",
    "random.seed(SEED)\n",
    "torch.manual_seed(SEED)\n",
    "torch.cuda.manual_seed(SEED)\n",
    "torch.backends.cudnn.deterministic=True\n",
    "torch.backends.cudnn.benchmark=False\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "'''\n",
    "编码器Encoder的实现\n",
    "'''\n",
    "class Encoder(nn.Module):\n",
    "    def __init__(self, input_dim, emb_dim, intent_dim, hid_dim, n_layers, kernel_size, dropout, max_length=50):\n",
    "        super(Encoder, self).__init__()\n",
    "        \n",
    "        assert kernel_size % 2 == 1,'kernel size must be odd!' # 卷积核size为奇数，方便序列两边pad处理\n",
    "        \n",
    "        self.scale = torch.sqrt(torch.FloatTensor([0.5])).to(device) # 确保整个网络的方差不会发生显著变化\n",
    "        \n",
    "        self.tok_embedding = nn.Embedding(input_dim, emb_dim) # token编码\n",
    "        self.pos_embedding = nn.Embedding(max_length, emb_dim) # token的位置编码\n",
    "        \n",
    "        self.emb2hid = nn.Linear(emb_dim, hid_dim) # 线性层，从emb_dim转为hid_dim\n",
    "        self.hid2emb = nn.Linear(hid_dim, emb_dim) # 线性层，从hid_dim转为emb_dim\n",
    "        \n",
    "        # 卷积块\n",
    "        self.convs = nn.ModuleList([nn.Conv1d(in_channels=hid_dim,\n",
    "                                              out_channels=2*hid_dim, # 卷积后输出的维度，这里2*hid_dim是为了后面的glu激活函数\n",
    "                                              kernel_size=kernel_size,\n",
    "                                              padding=(kernel_size - 1)//2) # 序列两边补0个数，保持维度不变\n",
    "                                              for _ in range(n_layers)]) \n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "        \n",
    "        # 利用encoder的输出进行意图识别\n",
    "        self.intent_output = nn.Linear(emb_dim, intent_dim)\n",
    "        \n",
    "    def forward(self, src):\n",
    "        # src: [batch_size, src_len]\n",
    "        batch_size = src.shape[0]\n",
    "        src_len = src.shape[1]\n",
    "        \n",
    "        # 创建token位置信息\n",
    "        pos = torch.arange(src_len).unsqueeze(0).repeat(batch_size, 1).to(device) # [batch_size, src_len]\n",
    "        \n",
    "        # 对token与其位置进行编码\n",
    "        tok_embedded = self.tok_embedding(src) # [batch_size, src_len, emb_dim]\n",
    "        pos_embedded = self.pos_embedding(pos.long()) # [batch_size, src_len, emb_dim]\n",
    "        \n",
    "        # 对token embedded和pos_embedded逐元素加和\n",
    "        embedded = self.dropout(tok_embedded + pos_embedded) # [batch_size, src_len, emb_dim]\n",
    "        \n",
    "        # embedded经过一线性层，将emb_dim转为hid_dim，作为卷积块的输入\n",
    "        conv_input = self.emb2hid(embedded) # [batch_size, src_len, hid_dim]\n",
    "        \n",
    "        # 转变维度，卷积在输入数据的最后一维进行\n",
    "        conv_input = conv_input.permute(0, 2, 1) # [batch_size, hid_dim, src_len]\n",
    "        \n",
    "        # 以下进行卷积块\n",
    "        for i, conv in enumerate(self.convs):\n",
    "            # 进行卷积\n",
    "            conved = conv(self.dropout(conv_input)) # [batch_size, 2*hid_dim, src_len]\n",
    "            \n",
    "            # 进行激活glu\n",
    "            conved = F.glu(conved, dim=1) # [batch_size, hid_dim, src_len]\n",
    "            \n",
    "            # 进行残差连接\n",
    "            conved = (conved + conv_input) * self.scale # [batch_size, hid_dim, src_len]\n",
    "            \n",
    "            # 作为下一个卷积块的输入\n",
    "            conv_input = conved\n",
    "        \n",
    "        # 经过一线性层，将hid_dim转为emb_dim，作为enocder的卷积输出的特征\n",
    "        conved = self.hid2emb(conved.permute(0, 2, 1)) # [batch_size, src_len, emb_dim]\n",
    "        \n",
    "        # 又是一个残差连接，逐元素加和输出，作为encoder的联合输出特征\n",
    "        combined = (conved + embedded) * self.scale # [batch_size, src_len, emb_dim]\n",
    "        \n",
    "        # 意图识别,加一个平均池化,池化后的维度是：[batch_size, emb_dim]\n",
    "        intent_output = self.intent_output(F.avg_pool1d(combined.permute(0, 2, 1), combined.shape[1]).squeeze()) # [batch_size, intent_dim]\n",
    "        \n",
    "        return conved, combined, intent_output\n",
    "    \n",
    "'''\n",
    "解码器Decoder实现\n",
    "'''\n",
    "class Decoder(nn.Module):\n",
    "    def __init__(self, output_dim, emb_dim, hid_dim, n_layers,kernel_size, dropout, trg_pad_idx, max_length=50):\n",
    "        super(Decoder, self).__init__()\n",
    "        self.kernel_size = kernel_size\n",
    "        self.trg_pad_idx = trg_pad_idx\n",
    "        \n",
    "        self.scale = torch.sqrt(torch.FloatTensor([0.5])).to(device)\n",
    "        \n",
    "        self.tok_embedding = nn.Embedding(output_dim, emb_dim)\n",
    "        self.pos_embedding = nn.Embedding(max_length, emb_dim)\n",
    "        \n",
    "        self.emb2hid = nn.Linear(emb_dim, hid_dim)\n",
    "        self.hid2emb = nn.Linear(hid_dim, emb_dim)\n",
    "        \n",
    "        self.attn_hid2emb = nn.Linear(hid_dim, emb_dim)\n",
    "        self.attn_emb2hid = nn.Linear(emb_dim, hid_dim)\n",
    "        \n",
    "        self.fc_out = nn.Linear(emb_dim, output_dim)\n",
    "        \n",
    "        self.convs = nn.ModuleList([nn.Conv1d(in_channels=hid_dim,\n",
    "                                              out_channels=2*hid_dim,\n",
    "                                              kernel_size=kernel_size)\n",
    "                                              for _ in range(n_layers)])\n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "        \n",
    "    def calculate_attention(self, embedded, conved, encoder_conved, encoder_combined):\n",
    "        '''\n",
    "        embedded:[batch_size, trg_Len, emb_dim]\n",
    "        conved:[batch_size, hid_dim, trg_len]\n",
    "        encoder_conved:[batch_size, src_len, emb_dim]\n",
    "        encoder_combined:[batch_size, src_len, emb_dim]\n",
    "        '''\n",
    "        # 经过一线性层，将hid_dim转为emb_dim，作为deocder的卷积输出的特征\n",
    "        conved_emb = self.attn_hid2emb(conved.permute(0, 2, 1)) # [batch_size, trg_len, emb_dim]\n",
    "        \n",
    "        # 一个残差连接，逐元素加和输出，作为decoder的联合输出特征\n",
    "        combined = (conved_emb + embedded) * self.scale # [batch_size, trg_len, emb_dim]\n",
    "        \n",
    "        # decoder的联合特征combined与encoder的卷积输出进行矩阵相乘\n",
    "        energy = torch.matmul(combined, encoder_conved.permute(0, 2, 1)) # [batch_size, trg_len, src_len]\n",
    "        \n",
    "        attention = F.softmax(energy, dim=2) # [batch_size, trg_len, src_len]\n",
    "        \n",
    "        attention_encoding = torch.matmul(attention, encoder_combined) # [batch_size, trg_len, emb_dim]\n",
    "        \n",
    "        # 经过一线性层，将emb_dim转为hid_dim\n",
    "        attended_encoding = self.attn_emb2hid(attention_encoding) # [batch_size, trg_len, hid_dim]\n",
    "        \n",
    "        # 一个残差连接，逐元素加和输出\n",
    "        attended_combined = (conved + attended_encoding.permute(0, 2, 1)) * self.scale # [batch_size, hid_dim, trg_len]\n",
    "        \n",
    "        return attention, attended_combined\n",
    "    \n",
    "    def forward(self, trg, encoder_conved, encoder_combined):\n",
    "        '''\n",
    "        trg:[batch_size, trg_len]\n",
    "        encoder_conved:[batch_size, src_len, emb_dim]\n",
    "        encoder_combined:[batch_size, src_len, emb_dim]\n",
    "        '''\n",
    "        batch_size = trg.shape[0]\n",
    "        trg_len = trg.shape[1]\n",
    "        \n",
    "        # 位置编码\n",
    "        pos = torch.arange(trg_len).unsqueeze(0).repeat(batch_size, 1).to(device) # [batch_size, trg_len]\n",
    "        \n",
    "        # 对token和pos进行embedding\n",
    "        tok_embedded = self.tok_embedding(trg) # [batch_size, trg_len, emb_dim]\n",
    "        pos_embedded = self.pos_embedding(pos.long()) # [batch_size, trg_len, emb_dim]\n",
    "        \n",
    "        # 对token embedded和pos_embedded逐元素加和\n",
    "        embedded = self.dropout(tok_embedded + pos_embedded) # [batch_size, trg_len, emb_dim]\n",
    "        \n",
    "        # 经过一线性层，将emb_dim转为hid_dim，作为卷积的输入\n",
    "        conv_input = self.emb2hid(embedded) # [batch_size, trg_len, hid_dim]\n",
    "        \n",
    "        # 转变维度，卷积在输入数据的最后一维进行\n",
    "        conv_input = conv_input.permute(0, 2, 1) # [batch_size, hid_dim, trg_len]\n",
    "        \n",
    "        batch_size = conv_input.shape[0]\n",
    "        hid_dim = conv_input.shape[1]\n",
    "        \n",
    "        # 卷积块\n",
    "        for i, conv in enumerate(self.convs):\n",
    "            conv_input = self.dropout(conv_input)\n",
    "            \n",
    "            # 在序列的一端进行pad\n",
    "            padding = torch.zeros(batch_size, hid_dim, self.kernel_size - 1).fill_(self.trg_pad_idx).to(device)\n",
    "            \n",
    "            padded_conv_input = torch.cat((padding, conv_input), dim=2) # [batch_size, hid_dim, trg_len + kernel_size - 1]\n",
    "            \n",
    "            # 进行卷积\n",
    "            conved = conv(padded_conv_input) # [batch_size, 2 * hid_dim, trg_len]\n",
    "            \n",
    "            # 经过glu激活\n",
    "            conved = F.glu(conved, dim=1) # [batch_size, hid_dim, trg_len]\n",
    "            \n",
    "            # 计算attention\n",
    "            attention, conved = self.calculate_attention(embedded, conved, encoder_conved, encoder_combined) # [batch_size, trg_len, src_len], [batch_size, hid_dim, trg_len]\n",
    "            \n",
    "            # 残差连接\n",
    "            conved = (conved + conv_input) * self.scale # [batch_size, hid_dim, trg_len]\n",
    "            \n",
    "            # 作为下一层卷积的输入\n",
    "            conv_input = conved\n",
    "        \n",
    "        conved = self.hid2emb(conved.permute(0, 2, 1)) # [batch_size, trg_len, emb_dim]\n",
    "        \n",
    "        # 预测输出\n",
    "        output = self.fc_out(self.dropout(conved)) # [batch_size, trg_len, output_dim]\n",
    "        \n",
    "        return output, attention\n",
    "    \n",
    "# 包装Encoder与Decoer\n",
    "class Seq2Seq(nn.Module):\n",
    "    def __init__(self, encoder, decoder):\n",
    "        super(Seq2Seq, self).__init__()\n",
    "        \n",
    "        # 编码器\n",
    "        self.encoder = encoder\n",
    "        \n",
    "        # 解码器用于slot槽识别\n",
    "        self.decoder = decoder\n",
    "        \n",
    "    def forward(self, src, trg):\n",
    "        '''\n",
    "        src:[batch_size, src_len]\n",
    "        trg:[batch_size, trg_Len-1] # decoder的输入去除了<eos>\n",
    "        \n",
    "        encoder_conved是encoder中最后一个卷积层的输出\n",
    "        encoder_combined是encoder_conved + (src_embedding + postional_embedding)\n",
    "        '''\n",
    "        encoder_conved, encoder_combined, intent_output = self.encoder(src) # [batch_size, src_len, emb_dim]; [batch_size, src_len, emb_dim]\n",
    "        \n",
    "        # decoder是对一批数据进行预测输出\n",
    "        slot_output, attention = self.decoder(trg, encoder_conved, encoder_combined) # [batch_size, trg_len-1, output_dim]; [batch_size, trg_len-1, src_len]\n",
    "        \n",
    "        return intent_output, slot_output, attention"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "'''\n",
    "定义seq2seq model\n",
    "'''\n",
    "input_dim = len(SOURCE.vocab)\n",
    "output_dim = len(TARGET.vocab) # slot size\n",
    "intent_dim = len(LABEL.vocab) # intent size\n",
    "emb_dim = 64\n",
    "hid_dim = 32\n",
    "enc_layers = 5 # encoder中几层卷积块\n",
    "dec_layers = 5 # decoder中几层卷积块\n",
    "enc_kernel_size = 3\n",
    "dec_kernel_size = 3\n",
    "enc_dropout = 0.25\n",
    "dec_dropout = 0.25\n",
    "trg_pad_idx = TARGET.vocab.stoi[TARGET.pad_token]\n",
    "\n",
    "enc = Encoder(input_dim, emb_dim, intent_dim, hid_dim, enc_layers, enc_kernel_size, enc_dropout)\n",
    "dec = Decoder(output_dim, emb_dim, hid_dim, dec_layers, dec_kernel_size, dec_dropout, trg_pad_idx)\n",
    "\n",
    "model = Seq2Seq(enc, dec).to(device)\n",
    "\n",
    "# 优化函数\n",
    "optimizer = optim.Adam(model.parameters())\n",
    "\n",
    "# 损失函数(slot)\n",
    "loss_slot = nn.CrossEntropyLoss(ignore_index=trg_pad_idx)\n",
    "\n",
    "# 定义损失函数(意图识别)\n",
    "loss_intent = nn.CrossEntropyLoss()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 训练\n",
    "def train(model, iterator, optimizer, loss_slot, loss_intent, clip):\n",
    "    \n",
    "    model.train()\n",
    "    epoch_loss = 0\n",
    "    \n",
    "    for i, batch in enumerate(iterator):\n",
    "        \n",
    "        src, _ = batch.source  # src=[batch_size, seq_len]，这里batch.src返回src和src的长度，因为在使用torchtext.Field时设置include_lengths=True\n",
    "        trg, _ = batch.target  # trg=[batch_size, seq_len]\n",
    "        label = batch.intent # [batch_size]\n",
    "        src = src.to(device)\n",
    "        trg = trg.to(device)\n",
    "        label = label.to(device)\n",
    "        \n",
    "        optimizer.zero_grad()\n",
    "        \n",
    "        intent_output, slot_output, _ = model(src, trg[:,:-1]) # [batch_size, intent_dim]; [batch_size, trg_len-1, slot_output_dim]\n",
    "        \n",
    "        # 1.计算slot loss\n",
    "        slot_output_dim = slot_output.shape[-1]\n",
    "        \n",
    "        slot_output = slot_output.contiguous().view(-1, slot_output_dim) # [batch_size * (trg_len-1), slot_output_dim]\n",
    "        \n",
    "        trg = trg[:,1:].contiguous().view(-1) # [batch_size * (trg_len-1)]\n",
    "        \n",
    "        loss1 = loss_slot(slot_output, trg)\n",
    "        \n",
    "        # 2.计算intent loss\n",
    "        loss2 = loss_intent(intent_output, label)\n",
    "        \n",
    "        loss = loss1 + loss2\n",
    "        \n",
    "        loss.backward()\n",
    "        \n",
    "        torch.nn.utils.clip_grad_norm_(model.parameters(), clip)\n",
    "        \n",
    "        optimizer.step()\n",
    "        \n",
    "        epoch_loss += loss.item()\n",
    "    \n",
    "    return epoch_loss / len(iterator)\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# val loss\n",
    "def evaluate(model, iterator, loss_slot, loss_intent):\n",
    "    model.eval()\n",
    "    \n",
    "    epoch_loss = 0\n",
    "    \n",
    "    with torch.no_grad():\n",
    "        for i, batch in enumerate(iterator):\n",
    "            src, _ = batch.source  # src=[batch_size, seq_len]\n",
    "            trg, _ = batch.target  # trg=[batch_size, seq_len]\n",
    "            label = batch.intent\n",
    "            src = src.to(device)\n",
    "            trg = trg.to(device)\n",
    "            label = label.to(device)\n",
    "            \n",
    "            intent_output, slot_output, _ = model(src, trg[:,:-1]) # [batch_size, intent_dim]; [batch_size, trg_len-1, slot_output_dim]\n",
    "            \n",
    "            # 1.计算slot loss\n",
    "            slot_output_dim = slot_output.shape[-1]\n",
    "\n",
    "            slot_output = slot_output.contiguous().view(-1, slot_output_dim) # [batch_size * (trg_len-1), slot_output_dim]\n",
    "\n",
    "            trg = trg[:,1:].contiguous().view(-1) # [batch_size * (trg_len-1)]\n",
    "\n",
    "            loss1 = loss_slot(slot_output, trg)\n",
    "\n",
    "            # 2.计算intent loss\n",
    "            loss2 = loss_intent(intent_output, label)\n",
    "\n",
    "            loss = loss1 + loss2\n",
    "        \n",
    "            \n",
    "            epoch_loss += loss.item()\n",
    "            \n",
    "    return epoch_loss / len(iterator)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def epoch_time(start_time, end_time):\n",
    "    elapsed_time = end_time - start_time\n",
    "    elapsed_mins = int(elapsed_time / 60)\n",
    "    elapsed_secs = int(elapsed_time - (elapsed_mins * 60))\n",
    "    return elapsed_mins, elapsed_secs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 01 | Time: 0m 4s\n",
      "\tTrain Loss: 3.795 | Train PPL:  44.490\n",
      "\t Val. Loss: 3.148 |  Val. PPL:  23.286\n",
      "Epoch: 02 | Time: 0m 3s\n",
      "\tTrain Loss: 2.716 | Train PPL:  15.125\n",
      "\t Val. Loss: 2.849 |  Val. PPL:  17.278\n",
      "Epoch: 03 | Time: 0m 2s\n",
      "\tTrain Loss: 2.541 | Train PPL:  12.696\n",
      "\t Val. Loss: 2.742 |  Val. PPL:  15.512\n",
      "Epoch: 04 | Time: 0m 2s\n",
      "\tTrain Loss: 2.325 | Train PPL:  10.231\n",
      "\t Val. Loss: 2.397 |  Val. PPL:  10.993\n",
      "Epoch: 05 | Time: 0m 2s\n",
      "\tTrain Loss: 2.085 | Train PPL:   8.047\n",
      "\t Val. Loss: 2.232 |  Val. PPL:   9.320\n",
      "Epoch: 06 | Time: 0m 2s\n",
      "\tTrain Loss: 1.978 | Train PPL:   7.227\n",
      "\t Val. Loss: 2.121 |  Val. PPL:   8.335\n",
      "Epoch: 07 | Time: 0m 2s\n",
      "\tTrain Loss: 1.907 | Train PPL:   6.731\n",
      "\t Val. Loss: 2.050 |  Val. PPL:   7.766\n",
      "Epoch: 08 | Time: 0m 3s\n",
      "\tTrain Loss: 1.840 | Train PPL:   6.299\n",
      "\t Val. Loss: 2.004 |  Val. PPL:   7.420\n",
      "Epoch: 09 | Time: 0m 3s\n",
      "\tTrain Loss: 1.784 | Train PPL:   5.954\n",
      "\t Val. Loss: 1.968 |  Val. PPL:   7.160\n",
      "Epoch: 10 | Time: 0m 2s\n",
      "\tTrain Loss: 1.742 | Train PPL:   5.707\n",
      "\t Val. Loss: 1.941 |  Val. PPL:   6.968\n",
      "Epoch: 11 | Time: 0m 2s\n",
      "\tTrain Loss: 1.715 | Train PPL:   5.555\n",
      "\t Val. Loss: 1.862 |  Val. PPL:   6.438\n",
      "Epoch: 12 | Time: 0m 3s\n",
      "\tTrain Loss: 1.669 | Train PPL:   5.305\n",
      "\t Val. Loss: 1.836 |  Val. PPL:   6.269\n",
      "Epoch: 13 | Time: 0m 3s\n",
      "\tTrain Loss: 1.625 | Train PPL:   5.076\n",
      "\t Val. Loss: 1.786 |  Val. PPL:   5.966\n",
      "Epoch: 14 | Time: 0m 3s\n",
      "\tTrain Loss: 1.606 | Train PPL:   4.982\n",
      "\t Val. Loss: 1.753 |  Val. PPL:   5.773\n",
      "Epoch: 15 | Time: 0m 3s\n",
      "\tTrain Loss: 1.554 | Train PPL:   4.730\n",
      "\t Val. Loss: 1.743 |  Val. PPL:   5.714\n",
      "Epoch: 16 | Time: 0m 2s\n",
      "\tTrain Loss: 1.535 | Train PPL:   4.642\n",
      "\t Val. Loss: 1.732 |  Val. PPL:   5.653\n",
      "Epoch: 17 | Time: 0m 2s\n",
      "\tTrain Loss: 1.515 | Train PPL:   4.548\n",
      "\t Val. Loss: 1.719 |  Val. PPL:   5.579\n",
      "Epoch: 18 | Time: 0m 2s\n",
      "\tTrain Loss: 1.498 | Train PPL:   4.471\n",
      "\t Val. Loss: 1.677 |  Val. PPL:   5.349\n",
      "Epoch: 19 | Time: 0m 2s\n",
      "\tTrain Loss: 1.462 | Train PPL:   4.316\n",
      "\t Val. Loss: 1.681 |  Val. PPL:   5.369\n",
      "Epoch: 20 | Time: 0m 3s\n",
      "\tTrain Loss: 1.439 | Train PPL:   4.217\n",
      "\t Val. Loss: 1.645 |  Val. PPL:   5.181\n",
      "Epoch: 21 | Time: 0m 2s\n",
      "\tTrain Loss: 1.422 | Train PPL:   4.145\n",
      "\t Val. Loss: 1.631 |  Val. PPL:   5.108\n",
      "Epoch: 22 | Time: 0m 2s\n",
      "\tTrain Loss: 1.403 | Train PPL:   4.068\n",
      "\t Val. Loss: 1.614 |  Val. PPL:   5.021\n",
      "Epoch: 23 | Time: 0m 2s\n",
      "\tTrain Loss: 1.378 | Train PPL:   3.966\n",
      "\t Val. Loss: 1.609 |  Val. PPL:   4.997\n",
      "Epoch: 24 | Time: 0m 3s\n",
      "\tTrain Loss: 1.365 | Train PPL:   3.916\n",
      "\t Val. Loss: 1.591 |  Val. PPL:   4.910\n",
      "Epoch: 25 | Time: 0m 3s\n",
      "\tTrain Loss: 1.353 | Train PPL:   3.869\n",
      "\t Val. Loss: 1.573 |  Val. PPL:   4.819\n",
      "Epoch: 26 | Time: 0m 3s\n",
      "\tTrain Loss: 1.330 | Train PPL:   3.782\n",
      "\t Val. Loss: 1.543 |  Val. PPL:   4.679\n",
      "Epoch: 27 | Time: 0m 3s\n",
      "\tTrain Loss: 1.319 | Train PPL:   3.739\n",
      "\t Val. Loss: 1.547 |  Val. PPL:   4.699\n",
      "Epoch: 28 | Time: 0m 3s\n",
      "\tTrain Loss: 1.294 | Train PPL:   3.646\n",
      "\t Val. Loss: 1.525 |  Val. PPL:   4.596\n",
      "Epoch: 29 | Time: 0m 3s\n",
      "\tTrain Loss: 1.282 | Train PPL:   3.605\n",
      "\t Val. Loss: 1.518 |  Val. PPL:   4.565\n",
      "Epoch: 30 | Time: 0m 2s\n",
      "\tTrain Loss: 1.271 | Train PPL:   3.565\n",
      "\t Val. Loss: 1.515 |  Val. PPL:   4.549\n",
      "Epoch: 31 | Time: 0m 2s\n",
      "\tTrain Loss: 1.262 | Train PPL:   3.531\n",
      "\t Val. Loss: 1.466 |  Val. PPL:   4.330\n",
      "Epoch: 32 | Time: 0m 2s\n",
      "\tTrain Loss: 1.240 | Train PPL:   3.457\n",
      "\t Val. Loss: 1.453 |  Val. PPL:   4.276\n",
      "Epoch: 33 | Time: 0m 2s\n",
      "\tTrain Loss: 1.236 | Train PPL:   3.442\n",
      "\t Val. Loss: 1.468 |  Val. PPL:   4.340\n",
      "Epoch: 34 | Time: 0m 3s\n",
      "\tTrain Loss: 1.219 | Train PPL:   3.384\n",
      "\t Val. Loss: 1.410 |  Val. PPL:   4.096\n",
      "Epoch: 35 | Time: 0m 2s\n",
      "\tTrain Loss: 1.208 | Train PPL:   3.348\n",
      "\t Val. Loss: 1.452 |  Val. PPL:   4.274\n",
      "Epoch: 36 | Time: 0m 2s\n",
      "\tTrain Loss: 1.203 | Train PPL:   3.330\n",
      "\t Val. Loss: 1.420 |  Val. PPL:   4.136\n",
      "Epoch: 37 | Time: 0m 3s\n",
      "\tTrain Loss: 1.187 | Train PPL:   3.278\n",
      "\t Val. Loss: 1.378 |  Val. PPL:   3.968\n",
      "Epoch: 38 | Time: 0m 2s\n",
      "\tTrain Loss: 1.172 | Train PPL:   3.229\n",
      "\t Val. Loss: 1.385 |  Val. PPL:   3.994\n",
      "Epoch: 39 | Time: 0m 3s\n",
      "\tTrain Loss: 1.156 | Train PPL:   3.178\n",
      "\t Val. Loss: 1.430 |  Val. PPL:   4.179\n",
      "Epoch: 40 | Time: 0m 2s\n",
      "\tTrain Loss: 1.157 | Train PPL:   3.179\n",
      "\t Val. Loss: 1.366 |  Val. PPL:   3.920\n",
      "Epoch: 41 | Time: 0m 3s\n",
      "\tTrain Loss: 1.153 | Train PPL:   3.169\n",
      "\t Val. Loss: 1.421 |  Val. PPL:   4.139\n",
      "Epoch: 42 | Time: 0m 3s\n",
      "\tTrain Loss: 1.138 | Train PPL:   3.120\n",
      "\t Val. Loss: 1.373 |  Val. PPL:   3.946\n",
      "Epoch: 43 | Time: 0m 3s\n",
      "\tTrain Loss: 1.137 | Train PPL:   3.116\n",
      "\t Val. Loss: 1.371 |  Val. PPL:   3.941\n",
      "Epoch: 44 | Time: 0m 2s\n",
      "\tTrain Loss: 1.126 | Train PPL:   3.082\n",
      "\t Val. Loss: 1.342 |  Val. PPL:   3.827\n",
      "Epoch: 45 | Time: 0m 3s\n",
      "\tTrain Loss: 1.123 | Train PPL:   3.074\n",
      "\t Val. Loss: 1.324 |  Val. PPL:   3.758\n",
      "Epoch: 46 | Time: 0m 2s\n",
      "\tTrain Loss: 1.107 | Train PPL:   3.026\n",
      "\t Val. Loss: 1.334 |  Val. PPL:   3.795\n",
      "Epoch: 47 | Time: 0m 3s\n",
      "\tTrain Loss: 1.108 | Train PPL:   3.029\n",
      "\t Val. Loss: 1.319 |  Val. PPL:   3.738\n",
      "Epoch: 48 | Time: 0m 2s\n",
      "\tTrain Loss: 1.101 | Train PPL:   3.007\n",
      "\t Val. Loss: 1.291 |  Val. PPL:   3.637\n",
      "Epoch: 49 | Time: 0m 2s\n",
      "\tTrain Loss: 1.094 | Train PPL:   2.985\n",
      "\t Val. Loss: 1.316 |  Val. PPL:   3.730\n",
      "Epoch: 50 | Time: 0m 3s\n",
      "\tTrain Loss: 1.088 | Train PPL:   2.967\n",
      "\t Val. Loss: 1.291 |  Val. PPL:   3.636\n",
      "Epoch: 51 | Time: 0m 2s\n",
      "\tTrain Loss: 1.081 | Train PPL:   2.948\n",
      "\t Val. Loss: 1.303 |  Val. PPL:   3.681\n",
      "Epoch: 52 | Time: 0m 2s\n",
      "\tTrain Loss: 1.079 | Train PPL:   2.943\n",
      "\t Val. Loss: 1.293 |  Val. PPL:   3.644\n",
      "Epoch: 53 | Time: 0m 3s\n",
      "\tTrain Loss: 1.077 | Train PPL:   2.935\n",
      "\t Val. Loss: 1.303 |  Val. PPL:   3.679\n",
      "Epoch: 54 | Time: 0m 3s\n",
      "\tTrain Loss: 1.062 | Train PPL:   2.893\n",
      "\t Val. Loss: 1.272 |  Val. PPL:   3.570\n",
      "Epoch: 55 | Time: 0m 3s\n",
      "\tTrain Loss: 1.060 | Train PPL:   2.886\n",
      "\t Val. Loss: 1.272 |  Val. PPL:   3.568\n",
      "Epoch: 56 | Time: 0m 2s\n",
      "\tTrain Loss: 1.056 | Train PPL:   2.876\n",
      "\t Val. Loss: 1.304 |  Val. PPL:   3.683\n",
      "Epoch: 57 | Time: 0m 2s\n",
      "\tTrain Loss: 1.049 | Train PPL:   2.855\n",
      "\t Val. Loss: 1.280 |  Val. PPL:   3.598\n",
      "Epoch: 58 | Time: 0m 2s\n",
      "\tTrain Loss: 1.049 | Train PPL:   2.856\n",
      "\t Val. Loss: 1.268 |  Val. PPL:   3.555\n",
      "Epoch: 59 | Time: 0m 3s\n",
      "\tTrain Loss: 1.043 | Train PPL:   2.836\n",
      "\t Val. Loss: 1.287 |  Val. PPL:   3.624\n",
      "Epoch: 60 | Time: 0m 3s\n",
      "\tTrain Loss: 1.039 | Train PPL:   2.825\n",
      "\t Val. Loss: 1.257 |  Val. PPL:   3.515\n",
      "Epoch: 61 | Time: 0m 2s\n",
      "\tTrain Loss: 1.037 | Train PPL:   2.822\n",
      "\t Val. Loss: 1.252 |  Val. PPL:   3.497\n",
      "Epoch: 62 | Time: 0m 3s\n",
      "\tTrain Loss: 1.027 | Train PPL:   2.792\n",
      "\t Val. Loss: 1.243 |  Val. PPL:   3.467\n",
      "Epoch: 63 | Time: 0m 2s\n",
      "\tTrain Loss: 1.019 | Train PPL:   2.770\n",
      "\t Val. Loss: 1.261 |  Val. PPL:   3.530\n",
      "Epoch: 64 | Time: 0m 3s\n",
      "\tTrain Loss: 1.028 | Train PPL:   2.796\n",
      "\t Val. Loss: 1.236 |  Val. PPL:   3.440\n",
      "Epoch: 65 | Time: 0m 3s\n",
      "\tTrain Loss: 1.024 | Train PPL:   2.783\n",
      "\t Val. Loss: 1.249 |  Val. PPL:   3.487\n",
      "Epoch: 66 | Time: 0m 3s\n",
      "\tTrain Loss: 1.019 | Train PPL:   2.771\n",
      "\t Val. Loss: 1.227 |  Val. PPL:   3.410\n",
      "Epoch: 67 | Time: 0m 3s\n",
      "\tTrain Loss: 1.010 | Train PPL:   2.746\n",
      "\t Val. Loss: 1.233 |  Val. PPL:   3.432\n",
      "Epoch: 68 | Time: 0m 2s\n",
      "\tTrain Loss: 1.016 | Train PPL:   2.762\n",
      "\t Val. Loss: 1.205 |  Val. PPL:   3.338\n",
      "Epoch: 69 | Time: 0m 2s\n",
      "\tTrain Loss: 0.994 | Train PPL:   2.701\n",
      "\t Val. Loss: 1.223 |  Val. PPL:   3.398\n",
      "Epoch: 70 | Time: 0m 2s\n",
      "\tTrain Loss: 1.003 | Train PPL:   2.727\n",
      "\t Val. Loss: 1.194 |  Val. PPL:   3.299\n",
      "Epoch: 71 | Time: 0m 2s\n",
      "\tTrain Loss: 0.991 | Train PPL:   2.693\n",
      "\t Val. Loss: 1.201 |  Val. PPL:   3.324\n",
      "Epoch: 72 | Time: 0m 2s\n",
      "\tTrain Loss: 0.984 | Train PPL:   2.676\n",
      "\t Val. Loss: 1.187 |  Val. PPL:   3.277\n",
      "Epoch: 73 | Time: 0m 3s\n",
      "\tTrain Loss: 0.987 | Train PPL:   2.682\n",
      "\t Val. Loss: 1.187 |  Val. PPL:   3.277\n",
      "Epoch: 74 | Time: 0m 3s\n",
      "\tTrain Loss: 0.979 | Train PPL:   2.663\n",
      "\t Val. Loss: 1.178 |  Val. PPL:   3.248\n",
      "Epoch: 75 | Time: 0m 2s\n",
      "\tTrain Loss: 0.978 | Train PPL:   2.660\n",
      "\t Val. Loss: 1.183 |  Val. PPL:   3.264\n",
      "Epoch: 76 | Time: 0m 3s\n",
      "\tTrain Loss: 0.963 | Train PPL:   2.620\n",
      "\t Val. Loss: 1.182 |  Val. PPL:   3.261\n",
      "Epoch: 77 | Time: 0m 3s\n",
      "\tTrain Loss: 0.964 | Train PPL:   2.623\n",
      "\t Val. Loss: 1.174 |  Val. PPL:   3.236\n",
      "Epoch: 78 | Time: 0m 3s\n",
      "\tTrain Loss: 0.962 | Train PPL:   2.616\n",
      "\t Val. Loss: 1.201 |  Val. PPL:   3.324\n",
      "Epoch: 79 | Time: 0m 2s\n",
      "\tTrain Loss: 0.958 | Train PPL:   2.606\n",
      "\t Val. Loss: 1.204 |  Val. PPL:   3.333\n",
      "Epoch: 80 | Time: 0m 3s\n",
      "\tTrain Loss: 0.956 | Train PPL:   2.602\n",
      "\t Val. Loss: 1.159 |  Val. PPL:   3.185\n",
      "Epoch: 81 | Time: 0m 3s\n",
      "\tTrain Loss: 0.961 | Train PPL:   2.615\n",
      "\t Val. Loss: 1.162 |  Val. PPL:   3.196\n",
      "Epoch: 82 | Time: 0m 2s\n",
      "\tTrain Loss: 0.948 | Train PPL:   2.582\n",
      "\t Val. Loss: 1.161 |  Val. PPL:   3.192\n",
      "Epoch: 83 | Time: 0m 2s\n",
      "\tTrain Loss: 0.941 | Train PPL:   2.562\n",
      "\t Val. Loss: 1.157 |  Val. PPL:   3.180\n",
      "Epoch: 84 | Time: 0m 3s\n",
      "\tTrain Loss: 0.945 | Train PPL:   2.573\n",
      "\t Val. Loss: 1.136 |  Val. PPL:   3.113\n",
      "Epoch: 85 | Time: 0m 2s\n",
      "\tTrain Loss: 0.938 | Train PPL:   2.555\n",
      "\t Val. Loss: 1.144 |  Val. PPL:   3.140\n",
      "Epoch: 86 | Time: 0m 3s\n",
      "\tTrain Loss: 0.942 | Train PPL:   2.564\n",
      "\t Val. Loss: 1.152 |  Val. PPL:   3.165\n",
      "Epoch: 87 | Time: 0m 2s\n",
      "\tTrain Loss: 0.939 | Train PPL:   2.558\n",
      "\t Val. Loss: 1.126 |  Val. PPL:   3.083\n",
      "Epoch: 88 | Time: 0m 3s\n",
      "\tTrain Loss: 0.930 | Train PPL:   2.534\n",
      "\t Val. Loss: 1.134 |  Val. PPL:   3.108\n",
      "Epoch: 89 | Time: 0m 2s\n",
      "\tTrain Loss: 0.917 | Train PPL:   2.502\n",
      "\t Val. Loss: 1.100 |  Val. PPL:   3.003\n",
      "Epoch: 90 | Time: 0m 2s\n",
      "\tTrain Loss: 0.914 | Train PPL:   2.494\n",
      "\t Val. Loss: 1.130 |  Val. PPL:   3.095\n",
      "Epoch: 91 | Time: 0m 3s\n",
      "\tTrain Loss: 0.917 | Train PPL:   2.502\n",
      "\t Val. Loss: 1.131 |  Val. PPL:   3.098\n",
      "Epoch: 92 | Time: 0m 3s\n",
      "\tTrain Loss: 0.912 | Train PPL:   2.490\n",
      "\t Val. Loss: 1.137 |  Val. PPL:   3.116\n",
      "Epoch: 93 | Time: 0m 3s\n",
      "\tTrain Loss: 0.910 | Train PPL:   2.484\n",
      "\t Val. Loss: 1.126 |  Val. PPL:   3.084\n",
      "Epoch: 94 | Time: 0m 3s\n",
      "\tTrain Loss: 0.899 | Train PPL:   2.457\n",
      "\t Val. Loss: 1.104 |  Val. PPL:   3.015\n",
      "Epoch: 95 | Time: 0m 3s\n",
      "\tTrain Loss: 0.908 | Train PPL:   2.480\n",
      "\t Val. Loss: 1.137 |  Val. PPL:   3.117\n",
      "Epoch: 96 | Time: 0m 3s\n",
      "\tTrain Loss: 0.906 | Train PPL:   2.474\n",
      "\t Val. Loss: 1.151 |  Val. PPL:   3.163\n",
      "Epoch: 97 | Time: 0m 3s\n",
      "\tTrain Loss: 0.899 | Train PPL:   2.457\n",
      "\t Val. Loss: 1.106 |  Val. PPL:   3.022\n",
      "Epoch: 98 | Time: 0m 3s\n",
      "\tTrain Loss: 0.890 | Train PPL:   2.436\n",
      "\t Val. Loss: 1.127 |  Val. PPL:   3.085\n",
      "Epoch: 99 | Time: 0m 3s\n",
      "\tTrain Loss: 0.884 | Train PPL:   2.421\n",
      "\t Val. Loss: 1.152 |  Val. PPL:   3.165\n",
      "Epoch: 100 | Time: 0m 2s\n",
      "\tTrain Loss: 0.888 | Train PPL:   2.430\n",
      "\t Val. Loss: 1.114 |  Val. PPL:   3.046\n",
      "Epoch: 101 | Time: 0m 2s\n",
      "\tTrain Loss: 0.887 | Train PPL:   2.427\n",
      "\t Val. Loss: 1.115 |  Val. PPL:   3.050\n",
      "Epoch: 102 | Time: 0m 3s\n",
      "\tTrain Loss: 0.876 | Train PPL:   2.402\n",
      "\t Val. Loss: 1.073 |  Val. PPL:   2.925\n",
      "Epoch: 103 | Time: 0m 2s\n",
      "\tTrain Loss: 0.881 | Train PPL:   2.413\n",
      "\t Val. Loss: 1.079 |  Val. PPL:   2.941\n",
      "Epoch: 104 | Time: 0m 3s\n",
      "\tTrain Loss: 0.881 | Train PPL:   2.414\n",
      "\t Val. Loss: 1.118 |  Val. PPL:   3.059\n",
      "Epoch: 105 | Time: 0m 2s\n",
      "\tTrain Loss: 0.880 | Train PPL:   2.411\n",
      "\t Val. Loss: 1.096 |  Val. PPL:   2.991\n",
      "Epoch: 106 | Time: 0m 2s\n",
      "\tTrain Loss: 0.869 | Train PPL:   2.384\n",
      "\t Val. Loss: 1.067 |  Val. PPL:   2.908\n",
      "Epoch: 107 | Time: 0m 3s\n",
      "\tTrain Loss: 0.855 | Train PPL:   2.351\n",
      "\t Val. Loss: 1.071 |  Val. PPL:   2.918\n",
      "Epoch: 108 | Time: 0m 3s\n",
      "\tTrain Loss: 0.868 | Train PPL:   2.382\n",
      "\t Val. Loss: 1.100 |  Val. PPL:   3.003\n",
      "Epoch: 109 | Time: 0m 3s\n",
      "\tTrain Loss: 0.860 | Train PPL:   2.362\n",
      "\t Val. Loss: 1.099 |  Val. PPL:   3.000\n",
      "Epoch: 110 | Time: 0m 3s\n",
      "\tTrain Loss: 0.864 | Train PPL:   2.372\n",
      "\t Val. Loss: 1.098 |  Val. PPL:   2.998\n",
      "Epoch: 111 | Time: 0m 2s\n",
      "\tTrain Loss: 0.860 | Train PPL:   2.363\n",
      "\t Val. Loss: 1.083 |  Val. PPL:   2.953\n",
      "Epoch: 112 | Time: 0m 2s\n",
      "\tTrain Loss: 0.856 | Train PPL:   2.354\n",
      "\t Val. Loss: 1.051 |  Val. PPL:   2.861\n",
      "Epoch: 113 | Time: 0m 2s\n",
      "\tTrain Loss: 0.851 | Train PPL:   2.343\n",
      "\t Val. Loss: 1.088 |  Val. PPL:   2.969\n",
      "Epoch: 114 | Time: 0m 2s\n",
      "\tTrain Loss: 0.847 | Train PPL:   2.333\n",
      "\t Val. Loss: 1.074 |  Val. PPL:   2.927\n",
      "Epoch: 115 | Time: 0m 3s\n",
      "\tTrain Loss: 0.850 | Train PPL:   2.340\n",
      "\t Val. Loss: 1.056 |  Val. PPL:   2.876\n",
      "Epoch: 116 | Time: 0m 3s\n",
      "\tTrain Loss: 0.843 | Train PPL:   2.324\n",
      "\t Val. Loss: 1.059 |  Val. PPL:   2.883\n",
      "Epoch: 117 | Time: 0m 2s\n",
      "\tTrain Loss: 0.831 | Train PPL:   2.295\n",
      "\t Val. Loss: 1.059 |  Val. PPL:   2.884\n",
      "Epoch: 118 | Time: 0m 3s\n",
      "\tTrain Loss: 0.830 | Train PPL:   2.293\n",
      "\t Val. Loss: 1.094 |  Val. PPL:   2.985\n",
      "Epoch: 119 | Time: 0m 3s\n",
      "\tTrain Loss: 0.831 | Train PPL:   2.296\n",
      "\t Val. Loss: 1.061 |  Val. PPL:   2.890\n",
      "Epoch: 120 | Time: 0m 3s\n",
      "\tTrain Loss: 0.831 | Train PPL:   2.296\n",
      "\t Val. Loss: 1.051 |  Val. PPL:   2.861\n",
      "Epoch: 121 | Time: 0m 3s\n",
      "\tTrain Loss: 0.821 | Train PPL:   2.274\n",
      "\t Val. Loss: 1.039 |  Val. PPL:   2.827\n",
      "Epoch: 122 | Time: 0m 3s\n",
      "\tTrain Loss: 0.829 | Train PPL:   2.290\n",
      "\t Val. Loss: 1.047 |  Val. PPL:   2.850\n",
      "Epoch: 123 | Time: 0m 3s\n",
      "\tTrain Loss: 0.820 | Train PPL:   2.271\n",
      "\t Val. Loss: 1.024 |  Val. PPL:   2.784\n",
      "Epoch: 124 | Time: 0m 3s\n",
      "\tTrain Loss: 0.821 | Train PPL:   2.272\n",
      "\t Val. Loss: 1.025 |  Val. PPL:   2.788\n",
      "Epoch: 125 | Time: 0m 3s\n",
      "\tTrain Loss: 0.823 | Train PPL:   2.278\n",
      "\t Val. Loss: 1.026 |  Val. PPL:   2.789\n",
      "Epoch: 126 | Time: 0m 3s\n",
      "\tTrain Loss: 0.818 | Train PPL:   2.267\n",
      "\t Val. Loss: 1.017 |  Val. PPL:   2.765\n",
      "Epoch: 127 | Time: 0m 2s\n",
      "\tTrain Loss: 0.813 | Train PPL:   2.255\n",
      "\t Val. Loss: 1.023 |  Val. PPL:   2.782\n",
      "Epoch: 128 | Time: 0m 3s\n",
      "\tTrain Loss: 0.809 | Train PPL:   2.245\n",
      "\t Val. Loss: 1.013 |  Val. PPL:   2.753\n",
      "Epoch: 129 | Time: 0m 3s\n",
      "\tTrain Loss: 0.809 | Train PPL:   2.245\n",
      "\t Val. Loss: 1.040 |  Val. PPL:   2.829\n",
      "Epoch: 130 | Time: 0m 2s\n",
      "\tTrain Loss: 0.805 | Train PPL:   2.236\n",
      "\t Val. Loss: 1.005 |  Val. PPL:   2.733\n",
      "Epoch: 131 | Time: 0m 2s\n",
      "\tTrain Loss: 0.802 | Train PPL:   2.230\n",
      "\t Val. Loss: 1.010 |  Val. PPL:   2.745\n",
      "Epoch: 132 | Time: 0m 2s\n",
      "\tTrain Loss: 0.802 | Train PPL:   2.230\n",
      "\t Val. Loss: 1.021 |  Val. PPL:   2.776\n",
      "Epoch: 133 | Time: 0m 3s\n",
      "\tTrain Loss: 0.786 | Train PPL:   2.194\n",
      "\t Val. Loss: 1.037 |  Val. PPL:   2.821\n",
      "Epoch: 134 | Time: 0m 2s\n",
      "\tTrain Loss: 0.787 | Train PPL:   2.196\n",
      "\t Val. Loss: 0.998 |  Val. PPL:   2.712\n",
      "Epoch: 135 | Time: 0m 2s\n",
      "\tTrain Loss: 0.782 | Train PPL:   2.186\n",
      "\t Val. Loss: 1.002 |  Val. PPL:   2.722\n",
      "Epoch: 136 | Time: 0m 2s\n",
      "\tTrain Loss: 0.781 | Train PPL:   2.183\n",
      "\t Val. Loss: 0.980 |  Val. PPL:   2.665\n",
      "Epoch: 137 | Time: 0m 3s\n",
      "\tTrain Loss: 0.777 | Train PPL:   2.174\n",
      "\t Val. Loss: 1.019 |  Val. PPL:   2.771\n",
      "Epoch: 138 | Time: 0m 2s\n",
      "\tTrain Loss: 0.775 | Train PPL:   2.172\n",
      "\t Val. Loss: 0.985 |  Val. PPL:   2.677\n",
      "Epoch: 139 | Time: 0m 2s\n",
      "\tTrain Loss: 0.766 | Train PPL:   2.152\n",
      "\t Val. Loss: 1.002 |  Val. PPL:   2.724\n",
      "Epoch: 140 | Time: 0m 2s\n",
      "\tTrain Loss: 0.766 | Train PPL:   2.151\n",
      "\t Val. Loss: 0.988 |  Val. PPL:   2.685\n",
      "Epoch: 141 | Time: 0m 3s\n",
      "\tTrain Loss: 0.759 | Train PPL:   2.136\n",
      "\t Val. Loss: 0.966 |  Val. PPL:   2.627\n",
      "Epoch: 142 | Time: 0m 3s\n",
      "\tTrain Loss: 0.769 | Train PPL:   2.158\n",
      "\t Val. Loss: 0.974 |  Val. PPL:   2.648\n",
      "Epoch: 143 | Time: 0m 3s\n",
      "\tTrain Loss: 0.748 | Train PPL:   2.113\n",
      "\t Val. Loss: 0.958 |  Val. PPL:   2.607\n",
      "Epoch: 144 | Time: 0m 3s\n",
      "\tTrain Loss: 0.746 | Train PPL:   2.108\n",
      "\t Val. Loss: 0.958 |  Val. PPL:   2.606\n",
      "Epoch: 145 | Time: 0m 2s\n",
      "\tTrain Loss: 0.753 | Train PPL:   2.123\n",
      "\t Val. Loss: 0.921 |  Val. PPL:   2.512\n",
      "Epoch: 146 | Time: 0m 3s\n",
      "\tTrain Loss: 0.746 | Train PPL:   2.109\n",
      "\t Val. Loss: 0.944 |  Val. PPL:   2.570\n",
      "Epoch: 147 | Time: 0m 3s\n",
      "\tTrain Loss: 0.734 | Train PPL:   2.083\n",
      "\t Val. Loss: 0.968 |  Val. PPL:   2.633\n",
      "Epoch: 148 | Time: 0m 3s\n",
      "\tTrain Loss: 0.731 | Train PPL:   2.078\n",
      "\t Val. Loss: 0.938 |  Val. PPL:   2.554\n",
      "Epoch: 149 | Time: 0m 3s\n",
      "\tTrain Loss: 0.734 | Train PPL:   2.084\n",
      "\t Val. Loss: 0.934 |  Val. PPL:   2.545\n",
      "Epoch: 150 | Time: 0m 3s\n",
      "\tTrain Loss: 0.730 | Train PPL:   2.075\n",
      "\t Val. Loss: 0.939 |  Val. PPL:   2.558\n",
      "Epoch: 151 | Time: 0m 3s\n",
      "\tTrain Loss: 0.727 | Train PPL:   2.070\n",
      "\t Val. Loss: 0.931 |  Val. PPL:   2.537\n",
      "Epoch: 152 | Time: 0m 3s\n",
      "\tTrain Loss: 0.714 | Train PPL:   2.042\n",
      "\t Val. Loss: 0.932 |  Val. PPL:   2.540\n",
      "Epoch: 153 | Time: 0m 3s\n",
      "\tTrain Loss: 0.714 | Train PPL:   2.043\n",
      "\t Val. Loss: 0.928 |  Val. PPL:   2.529\n",
      "Epoch: 154 | Time: 0m 3s\n",
      "\tTrain Loss: 0.704 | Train PPL:   2.021\n",
      "\t Val. Loss: 0.923 |  Val. PPL:   2.518\n",
      "Epoch: 155 | Time: 0m 3s\n",
      "\tTrain Loss: 0.704 | Train PPL:   2.022\n",
      "\t Val. Loss: 0.912 |  Val. PPL:   2.489\n",
      "Epoch: 156 | Time: 0m 2s\n",
      "\tTrain Loss: 0.701 | Train PPL:   2.015\n",
      "\t Val. Loss: 0.936 |  Val. PPL:   2.549\n",
      "Epoch: 157 | Time: 0m 3s\n",
      "\tTrain Loss: 0.694 | Train PPL:   2.003\n",
      "\t Val. Loss: 0.889 |  Val. PPL:   2.433\n",
      "Epoch: 158 | Time: 0m 2s\n",
      "\tTrain Loss: 0.703 | Train PPL:   2.019\n",
      "\t Val. Loss: 0.907 |  Val. PPL:   2.477\n",
      "Epoch: 159 | Time: 0m 2s\n",
      "\tTrain Loss: 0.686 | Train PPL:   1.985\n",
      "\t Val. Loss: 0.893 |  Val. PPL:   2.443\n",
      "Epoch: 160 | Time: 0m 2s\n",
      "\tTrain Loss: 0.681 | Train PPL:   1.975\n",
      "\t Val. Loss: 0.870 |  Val. PPL:   2.386\n",
      "Epoch: 161 | Time: 0m 3s\n",
      "\tTrain Loss: 0.688 | Train PPL:   1.989\n",
      "\t Val. Loss: 0.874 |  Val. PPL:   2.397\n",
      "Epoch: 162 | Time: 0m 2s\n",
      "\tTrain Loss: 0.672 | Train PPL:   1.958\n",
      "\t Val. Loss: 0.885 |  Val. PPL:   2.422\n",
      "Epoch: 163 | Time: 0m 2s\n",
      "\tTrain Loss: 0.677 | Train PPL:   1.967\n",
      "\t Val. Loss: 0.864 |  Val. PPL:   2.372\n",
      "Epoch: 164 | Time: 0m 2s\n",
      "\tTrain Loss: 0.669 | Train PPL:   1.953\n",
      "\t Val. Loss: 0.880 |  Val. PPL:   2.410\n",
      "Epoch: 165 | Time: 0m 3s\n",
      "\tTrain Loss: 0.661 | Train PPL:   1.937\n",
      "\t Val. Loss: 0.898 |  Val. PPL:   2.454\n",
      "Epoch: 166 | Time: 0m 2s\n",
      "\tTrain Loss: 0.652 | Train PPL:   1.920\n",
      "\t Val. Loss: 0.875 |  Val. PPL:   2.398\n",
      "Epoch: 167 | Time: 0m 3s\n",
      "\tTrain Loss: 0.649 | Train PPL:   1.913\n",
      "\t Val. Loss: 0.868 |  Val. PPL:   2.383\n",
      "Epoch: 168 | Time: 0m 3s\n",
      "\tTrain Loss: 0.636 | Train PPL:   1.888\n",
      "\t Val. Loss: 0.855 |  Val. PPL:   2.352\n",
      "Epoch: 169 | Time: 0m 3s\n",
      "\tTrain Loss: 0.633 | Train PPL:   1.883\n",
      "\t Val. Loss: 0.852 |  Val. PPL:   2.344\n",
      "Epoch: 170 | Time: 0m 3s\n",
      "\tTrain Loss: 0.633 | Train PPL:   1.884\n",
      "\t Val. Loss: 0.849 |  Val. PPL:   2.338\n",
      "Epoch: 171 | Time: 0m 3s\n",
      "\tTrain Loss: 0.622 | Train PPL:   1.862\n",
      "\t Val. Loss: 0.810 |  Val. PPL:   2.247\n",
      "Epoch: 172 | Time: 0m 2s\n",
      "\tTrain Loss: 0.617 | Train PPL:   1.852\n",
      "\t Val. Loss: 0.829 |  Val. PPL:   2.291\n",
      "Epoch: 173 | Time: 0m 2s\n",
      "\tTrain Loss: 0.609 | Train PPL:   1.839\n",
      "\t Val. Loss: 0.825 |  Val. PPL:   2.282\n",
      "Epoch: 174 | Time: 0m 3s\n",
      "\tTrain Loss: 0.609 | Train PPL:   1.839\n",
      "\t Val. Loss: 0.803 |  Val. PPL:   2.233\n",
      "Epoch: 175 | Time: 0m 3s\n",
      "\tTrain Loss: 0.607 | Train PPL:   1.836\n",
      "\t Val. Loss: 0.815 |  Val. PPL:   2.259\n",
      "Epoch: 176 | Time: 0m 3s\n",
      "\tTrain Loss: 0.605 | Train PPL:   1.831\n",
      "\t Val. Loss: 0.778 |  Val. PPL:   2.176\n",
      "Epoch: 177 | Time: 0m 3s\n",
      "\tTrain Loss: 0.593 | Train PPL:   1.810\n",
      "\t Val. Loss: 0.781 |  Val. PPL:   2.185\n",
      "Epoch: 178 | Time: 0m 2s\n",
      "\tTrain Loss: 0.591 | Train PPL:   1.805\n",
      "\t Val. Loss: 0.790 |  Val. PPL:   2.202\n",
      "Epoch: 179 | Time: 0m 2s\n",
      "\tTrain Loss: 0.581 | Train PPL:   1.787\n",
      "\t Val. Loss: 0.782 |  Val. PPL:   2.186\n",
      "Epoch: 180 | Time: 0m 2s\n",
      "\tTrain Loss: 0.577 | Train PPL:   1.781\n",
      "\t Val. Loss: 0.787 |  Val. PPL:   2.196\n",
      "Epoch: 181 | Time: 0m 3s\n",
      "\tTrain Loss: 0.567 | Train PPL:   1.764\n",
      "\t Val. Loss: 0.760 |  Val. PPL:   2.137\n",
      "Epoch: 182 | Time: 0m 2s\n",
      "\tTrain Loss: 0.565 | Train PPL:   1.759\n",
      "\t Val. Loss: 0.759 |  Val. PPL:   2.136\n",
      "Epoch: 183 | Time: 0m 2s\n",
      "\tTrain Loss: 0.549 | Train PPL:   1.731\n",
      "\t Val. Loss: 0.763 |  Val. PPL:   2.146\n",
      "Epoch: 184 | Time: 0m 2s\n",
      "\tTrain Loss: 0.550 | Train PPL:   1.734\n",
      "\t Val. Loss: 0.759 |  Val. PPL:   2.136\n",
      "Epoch: 185 | Time: 0m 3s\n",
      "\tTrain Loss: 0.546 | Train PPL:   1.726\n",
      "\t Val. Loss: 0.767 |  Val. PPL:   2.154\n",
      "Epoch: 186 | Time: 0m 3s\n",
      "\tTrain Loss: 0.542 | Train PPL:   1.720\n",
      "\t Val. Loss: 0.762 |  Val. PPL:   2.142\n",
      "Epoch: 187 | Time: 0m 2s\n",
      "\tTrain Loss: 0.541 | Train PPL:   1.717\n",
      "\t Val. Loss: 0.717 |  Val. PPL:   2.048\n",
      "Epoch: 188 | Time: 0m 3s\n",
      "\tTrain Loss: 0.528 | Train PPL:   1.695\n",
      "\t Val. Loss: 0.729 |  Val. PPL:   2.074\n",
      "Epoch: 189 | Time: 0m 2s\n",
      "\tTrain Loss: 0.519 | Train PPL:   1.681\n",
      "\t Val. Loss: 0.752 |  Val. PPL:   2.122\n",
      "Epoch: 190 | Time: 0m 2s\n",
      "\tTrain Loss: 0.520 | Train PPL:   1.683\n",
      "\t Val. Loss: 0.707 |  Val. PPL:   2.027\n",
      "Epoch: 191 | Time: 0m 3s\n",
      "\tTrain Loss: 0.520 | Train PPL:   1.682\n",
      "\t Val. Loss: 0.716 |  Val. PPL:   2.047\n",
      "Epoch: 192 | Time: 0m 3s\n",
      "\tTrain Loss: 0.506 | Train PPL:   1.658\n",
      "\t Val. Loss: 0.710 |  Val. PPL:   2.033\n",
      "Epoch: 193 | Time: 0m 3s\n",
      "\tTrain Loss: 0.495 | Train PPL:   1.640\n",
      "\t Val. Loss: 0.699 |  Val. PPL:   2.012\n",
      "Epoch: 194 | Time: 0m 3s\n",
      "\tTrain Loss: 0.497 | Train PPL:   1.644\n",
      "\t Val. Loss: 0.693 |  Val. PPL:   2.001\n",
      "Epoch: 195 | Time: 0m 3s\n",
      "\tTrain Loss: 0.487 | Train PPL:   1.628\n",
      "\t Val. Loss: 0.680 |  Val. PPL:   1.973\n",
      "Epoch: 196 | Time: 0m 2s\n",
      "\tTrain Loss: 0.472 | Train PPL:   1.604\n",
      "\t Val. Loss: 0.681 |  Val. PPL:   1.976\n",
      "Epoch: 197 | Time: 0m 2s\n",
      "\tTrain Loss: 0.478 | Train PPL:   1.613\n",
      "\t Val. Loss: 0.670 |  Val. PPL:   1.954\n",
      "Epoch: 198 | Time: 0m 3s\n",
      "\tTrain Loss: 0.461 | Train PPL:   1.586\n",
      "\t Val. Loss: 0.681 |  Val. PPL:   1.977\n",
      "Epoch: 199 | Time: 0m 2s\n",
      "\tTrain Loss: 0.448 | Train PPL:   1.565\n",
      "\t Val. Loss: 0.662 |  Val. PPL:   1.939\n",
      "Epoch: 200 | Time: 0m 2s\n",
      "\tTrain Loss: 0.449 | Train PPL:   1.566\n",
      "\t Val. Loss: 0.660 |  Val. PPL:   1.934\n",
      "Epoch: 201 | Time: 0m 3s\n",
      "\tTrain Loss: 0.437 | Train PPL:   1.547\n",
      "\t Val. Loss: 0.657 |  Val. PPL:   1.929\n",
      "Epoch: 202 | Time: 0m 2s\n",
      "\tTrain Loss: 0.437 | Train PPL:   1.548\n",
      "\t Val. Loss: 0.653 |  Val. PPL:   1.922\n",
      "Epoch: 203 | Time: 0m 2s\n",
      "\tTrain Loss: 0.420 | Train PPL:   1.521\n",
      "\t Val. Loss: 0.641 |  Val. PPL:   1.899\n",
      "Epoch: 204 | Time: 0m 3s\n",
      "\tTrain Loss: 0.410 | Train PPL:   1.507\n",
      "\t Val. Loss: 0.628 |  Val. PPL:   1.875\n",
      "Epoch: 205 | Time: 0m 2s\n",
      "\tTrain Loss: 0.410 | Train PPL:   1.507\n",
      "\t Val. Loss: 0.620 |  Val. PPL:   1.859\n",
      "Epoch: 206 | Time: 0m 3s\n",
      "\tTrain Loss: 0.404 | Train PPL:   1.498\n",
      "\t Val. Loss: 0.634 |  Val. PPL:   1.886\n",
      "Epoch: 207 | Time: 0m 3s\n",
      "\tTrain Loss: 0.406 | Train PPL:   1.500\n",
      "\t Val. Loss: 0.628 |  Val. PPL:   1.874\n",
      "Epoch: 208 | Time: 0m 2s\n",
      "\tTrain Loss: 0.395 | Train PPL:   1.484\n",
      "\t Val. Loss: 0.612 |  Val. PPL:   1.844\n",
      "Epoch: 209 | Time: 0m 2s\n",
      "\tTrain Loss: 0.395 | Train PPL:   1.484\n",
      "\t Val. Loss: 0.634 |  Val. PPL:   1.885\n",
      "Epoch: 210 | Time: 0m 2s\n",
      "\tTrain Loss: 0.385 | Train PPL:   1.470\n",
      "\t Val. Loss: 0.629 |  Val. PPL:   1.876\n",
      "Epoch: 211 | Time: 0m 3s\n",
      "\tTrain Loss: 0.376 | Train PPL:   1.456\n",
      "\t Val. Loss: 0.599 |  Val. PPL:   1.821\n",
      "Epoch: 212 | Time: 0m 3s\n",
      "\tTrain Loss: 0.376 | Train PPL:   1.457\n",
      "\t Val. Loss: 0.614 |  Val. PPL:   1.847\n",
      "Epoch: 213 | Time: 0m 3s\n",
      "\tTrain Loss: 0.369 | Train PPL:   1.446\n",
      "\t Val. Loss: 0.622 |  Val. PPL:   1.863\n",
      "Epoch: 214 | Time: 0m 3s\n",
      "\tTrain Loss: 0.364 | Train PPL:   1.439\n",
      "\t Val. Loss: 0.610 |  Val. PPL:   1.841\n",
      "Epoch: 215 | Time: 0m 3s\n",
      "\tTrain Loss: 0.363 | Train PPL:   1.438\n",
      "\t Val. Loss: 0.587 |  Val. PPL:   1.799\n",
      "Epoch: 216 | Time: 0m 2s\n",
      "\tTrain Loss: 0.351 | Train PPL:   1.421\n",
      "\t Val. Loss: 0.602 |  Val. PPL:   1.826\n",
      "Epoch: 217 | Time: 0m 2s\n",
      "\tTrain Loss: 0.355 | Train PPL:   1.426\n",
      "\t Val. Loss: 0.564 |  Val. PPL:   1.758\n",
      "Epoch: 218 | Time: 0m 3s\n",
      "\tTrain Loss: 0.350 | Train PPL:   1.420\n",
      "\t Val. Loss: 0.596 |  Val. PPL:   1.815\n",
      "Epoch: 219 | Time: 0m 2s\n",
      "\tTrain Loss: 0.343 | Train PPL:   1.410\n",
      "\t Val. Loss: 0.558 |  Val. PPL:   1.747\n",
      "Epoch: 220 | Time: 0m 3s\n",
      "\tTrain Loss: 0.332 | Train PPL:   1.394\n",
      "\t Val. Loss: 0.546 |  Val. PPL:   1.726\n",
      "Epoch: 221 | Time: 0m 3s\n",
      "\tTrain Loss: 0.337 | Train PPL:   1.401\n",
      "\t Val. Loss: 0.563 |  Val. PPL:   1.756\n",
      "Epoch: 222 | Time: 0m 3s\n",
      "\tTrain Loss: 0.331 | Train PPL:   1.393\n",
      "\t Val. Loss: 0.558 |  Val. PPL:   1.747\n",
      "Epoch: 223 | Time: 0m 3s\n",
      "\tTrain Loss: 0.331 | Train PPL:   1.392\n",
      "\t Val. Loss: 0.554 |  Val. PPL:   1.739\n",
      "Epoch: 224 | Time: 0m 3s\n",
      "\tTrain Loss: 0.322 | Train PPL:   1.379\n",
      "\t Val. Loss: 0.570 |  Val. PPL:   1.769\n",
      "Epoch: 225 | Time: 0m 2s\n",
      "\tTrain Loss: 0.316 | Train PPL:   1.371\n",
      "\t Val. Loss: 0.580 |  Val. PPL:   1.785\n",
      "Epoch: 226 | Time: 0m 2s\n",
      "\tTrain Loss: 0.313 | Train PPL:   1.368\n",
      "\t Val. Loss: 0.565 |  Val. PPL:   1.760\n",
      "Epoch: 227 | Time: 0m 3s\n",
      "\tTrain Loss: 0.312 | Train PPL:   1.367\n",
      "\t Val. Loss: 0.586 |  Val. PPL:   1.796\n",
      "Epoch: 228 | Time: 0m 3s\n",
      "\tTrain Loss: 0.313 | Train PPL:   1.368\n",
      "\t Val. Loss: 0.552 |  Val. PPL:   1.737\n",
      "Epoch: 229 | Time: 0m 2s\n",
      "\tTrain Loss: 0.310 | Train PPL:   1.364\n",
      "\t Val. Loss: 0.523 |  Val. PPL:   1.687\n",
      "Epoch: 230 | Time: 0m 2s\n",
      "\tTrain Loss: 0.302 | Train PPL:   1.353\n",
      "\t Val. Loss: 0.566 |  Val. PPL:   1.762\n",
      "Epoch: 231 | Time: 0m 3s\n",
      "\tTrain Loss: 0.294 | Train PPL:   1.342\n",
      "\t Val. Loss: 0.545 |  Val. PPL:   1.724\n",
      "Epoch: 232 | Time: 0m 3s\n",
      "\tTrain Loss: 0.298 | Train PPL:   1.347\n",
      "\t Val. Loss: 0.539 |  Val. PPL:   1.715\n",
      "Epoch: 233 | Time: 0m 3s\n",
      "\tTrain Loss: 0.295 | Train PPL:   1.343\n",
      "\t Val. Loss: 0.540 |  Val. PPL:   1.716\n",
      "Epoch: 234 | Time: 0m 2s\n",
      "\tTrain Loss: 0.292 | Train PPL:   1.339\n",
      "\t Val. Loss: 0.534 |  Val. PPL:   1.707\n",
      "Epoch: 235 | Time: 0m 3s\n",
      "\tTrain Loss: 0.292 | Train PPL:   1.339\n",
      "\t Val. Loss: 0.550 |  Val. PPL:   1.734\n",
      "Epoch: 236 | Time: 0m 2s\n",
      "\tTrain Loss: 0.281 | Train PPL:   1.324\n",
      "\t Val. Loss: 0.535 |  Val. PPL:   1.708\n",
      "Epoch: 237 | Time: 0m 2s\n",
      "\tTrain Loss: 0.284 | Train PPL:   1.328\n",
      "\t Val. Loss: 0.556 |  Val. PPL:   1.744\n",
      "Epoch: 238 | Time: 0m 2s\n",
      "\tTrain Loss: 0.281 | Train PPL:   1.324\n",
      "\t Val. Loss: 0.537 |  Val. PPL:   1.711\n",
      "Epoch: 239 | Time: 0m 2s\n",
      "\tTrain Loss: 0.278 | Train PPL:   1.320\n",
      "\t Val. Loss: 0.553 |  Val. PPL:   1.738\n",
      "Epoch: 240 | Time: 0m 3s\n",
      "\tTrain Loss: 0.282 | Train PPL:   1.326\n",
      "\t Val. Loss: 0.550 |  Val. PPL:   1.732\n",
      "Epoch: 241 | Time: 0m 3s\n",
      "\tTrain Loss: 0.270 | Train PPL:   1.310\n",
      "\t Val. Loss: 0.536 |  Val. PPL:   1.709\n",
      "Epoch: 242 | Time: 0m 3s\n",
      "\tTrain Loss: 0.268 | Train PPL:   1.307\n",
      "\t Val. Loss: 0.530 |  Val. PPL:   1.700\n",
      "Epoch: 243 | Time: 0m 3s\n",
      "\tTrain Loss: 0.269 | Train PPL:   1.309\n",
      "\t Val. Loss: 0.568 |  Val. PPL:   1.764\n",
      "Epoch: 244 | Time: 0m 3s\n",
      "\tTrain Loss: 0.275 | Train PPL:   1.317\n",
      "\t Val. Loss: 0.578 |  Val. PPL:   1.782\n",
      "Epoch: 245 | Time: 0m 2s\n",
      "\tTrain Loss: 0.266 | Train PPL:   1.304\n",
      "\t Val. Loss: 0.569 |  Val. PPL:   1.767\n",
      "Epoch: 246 | Time: 0m 2s\n",
      "\tTrain Loss: 0.260 | Train PPL:   1.297\n",
      "\t Val. Loss: 0.551 |  Val. PPL:   1.734\n",
      "Epoch: 247 | Time: 0m 3s\n",
      "\tTrain Loss: 0.261 | Train PPL:   1.299\n",
      "\t Val. Loss: 0.569 |  Val. PPL:   1.766\n",
      "Epoch: 248 | Time: 0m 3s\n",
      "\tTrain Loss: 0.261 | Train PPL:   1.298\n",
      "\t Val. Loss: 0.547 |  Val. PPL:   1.728\n",
      "Epoch: 249 | Time: 0m 3s\n",
      "\tTrain Loss: 0.254 | Train PPL:   1.290\n",
      "\t Val. Loss: 0.578 |  Val. PPL:   1.783\n",
      "Epoch: 250 | Time: 0m 3s\n",
      "\tTrain Loss: 0.253 | Train PPL:   1.287\n",
      "\t Val. Loss: 0.542 |  Val. PPL:   1.719\n",
      "Epoch: 251 | Time: 0m 2s\n",
      "\tTrain Loss: 0.246 | Train PPL:   1.278\n",
      "\t Val. Loss: 0.570 |  Val. PPL:   1.768\n",
      "Epoch: 252 | Time: 0m 2s\n",
      "\tTrain Loss: 0.250 | Train PPL:   1.283\n",
      "\t Val. Loss: 0.541 |  Val. PPL:   1.718\n",
      "Epoch: 253 | Time: 0m 3s\n",
      "\tTrain Loss: 0.246 | Train PPL:   1.279\n",
      "\t Val. Loss: 0.545 |  Val. PPL:   1.725\n",
      "Epoch: 254 | Time: 0m 3s\n",
      "\tTrain Loss: 0.239 | Train PPL:   1.270\n",
      "\t Val. Loss: 0.517 |  Val. PPL:   1.677\n",
      "Epoch: 255 | Time: 0m 2s\n",
      "\tTrain Loss: 0.244 | Train PPL:   1.277\n",
      "\t Val. Loss: 0.513 |  Val. PPL:   1.671\n",
      "Epoch: 256 | Time: 0m 3s\n",
      "\tTrain Loss: 0.238 | Train PPL:   1.269\n",
      "\t Val. Loss: 0.530 |  Val. PPL:   1.699\n",
      "Epoch: 257 | Time: 0m 3s\n",
      "\tTrain Loss: 0.246 | Train PPL:   1.279\n",
      "\t Val. Loss: 0.520 |  Val. PPL:   1.682\n",
      "Epoch: 258 | Time: 0m 3s\n",
      "\tTrain Loss: 0.235 | Train PPL:   1.265\n",
      "\t Val. Loss: 0.565 |  Val. PPL:   1.759\n",
      "Epoch: 259 | Time: 0m 2s\n",
      "\tTrain Loss: 0.236 | Train PPL:   1.266\n",
      "\t Val. Loss: 0.525 |  Val. PPL:   1.690\n",
      "Epoch: 260 | Time: 0m 2s\n",
      "\tTrain Loss: 0.228 | Train PPL:   1.256\n",
      "\t Val. Loss: 0.548 |  Val. PPL:   1.730\n",
      "Epoch: 261 | Time: 0m 3s\n",
      "\tTrain Loss: 0.236 | Train PPL:   1.266\n",
      "\t Val. Loss: 0.546 |  Val. PPL:   1.727\n",
      "Epoch: 262 | Time: 0m 2s\n",
      "\tTrain Loss: 0.231 | Train PPL:   1.260\n",
      "\t Val. Loss: 0.520 |  Val. PPL:   1.681\n",
      "Epoch: 263 | Time: 0m 3s\n",
      "\tTrain Loss: 0.235 | Train PPL:   1.265\n",
      "\t Val. Loss: 0.510 |  Val. PPL:   1.665\n",
      "Epoch: 264 | Time: 0m 3s\n",
      "\tTrain Loss: 0.229 | Train PPL:   1.257\n",
      "\t Val. Loss: 0.544 |  Val. PPL:   1.723\n",
      "Epoch: 265 | Time: 0m 3s\n",
      "\tTrain Loss: 0.224 | Train PPL:   1.251\n",
      "\t Val. Loss: 0.517 |  Val. PPL:   1.677\n",
      "Epoch: 266 | Time: 0m 3s\n",
      "\tTrain Loss: 0.222 | Train PPL:   1.248\n",
      "\t Val. Loss: 0.525 |  Val. PPL:   1.691\n",
      "Epoch: 267 | Time: 0m 2s\n",
      "\tTrain Loss: 0.228 | Train PPL:   1.256\n",
      "\t Val. Loss: 0.509 |  Val. PPL:   1.664\n",
      "Epoch: 268 | Time: 0m 3s\n",
      "\tTrain Loss: 0.219 | Train PPL:   1.244\n",
      "\t Val. Loss: 0.516 |  Val. PPL:   1.675\n",
      "Epoch: 269 | Time: 0m 3s\n",
      "\tTrain Loss: 0.214 | Train PPL:   1.238\n",
      "\t Val. Loss: 0.566 |  Val. PPL:   1.762\n",
      "Epoch: 270 | Time: 0m 3s\n",
      "\tTrain Loss: 0.215 | Train PPL:   1.240\n",
      "\t Val. Loss: 0.531 |  Val. PPL:   1.700\n",
      "Epoch: 271 | Time: 0m 2s\n",
      "\tTrain Loss: 0.217 | Train PPL:   1.242\n",
      "\t Val. Loss: 0.542 |  Val. PPL:   1.719\n",
      "Epoch: 272 | Time: 0m 3s\n",
      "\tTrain Loss: 0.214 | Train PPL:   1.239\n",
      "\t Val. Loss: 0.502 |  Val. PPL:   1.652\n",
      "Epoch: 273 | Time: 0m 2s\n",
      "\tTrain Loss: 0.213 | Train PPL:   1.237\n",
      "\t Val. Loss: 0.517 |  Val. PPL:   1.676\n",
      "Epoch: 274 | Time: 0m 3s\n",
      "\tTrain Loss: 0.207 | Train PPL:   1.230\n",
      "\t Val. Loss: 0.482 |  Val. PPL:   1.619\n",
      "Epoch: 275 | Time: 0m 2s\n",
      "\tTrain Loss: 0.213 | Train PPL:   1.237\n",
      "\t Val. Loss: 0.519 |  Val. PPL:   1.680\n",
      "Epoch: 276 | Time: 0m 3s\n",
      "\tTrain Loss: 0.211 | Train PPL:   1.235\n",
      "\t Val. Loss: 0.511 |  Val. PPL:   1.666\n",
      "Epoch: 277 | Time: 0m 3s\n",
      "\tTrain Loss: 0.208 | Train PPL:   1.232\n",
      "\t Val. Loss: 0.506 |  Val. PPL:   1.658\n",
      "Epoch: 278 | Time: 0m 2s\n",
      "\tTrain Loss: 0.203 | Train PPL:   1.225\n",
      "\t Val. Loss: 0.487 |  Val. PPL:   1.628\n",
      "Epoch: 279 | Time: 0m 3s\n",
      "\tTrain Loss: 0.202 | Train PPL:   1.224\n",
      "\t Val. Loss: 0.519 |  Val. PPL:   1.681\n",
      "Epoch: 280 | Time: 0m 2s\n",
      "\tTrain Loss: 0.206 | Train PPL:   1.228\n",
      "\t Val. Loss: 0.527 |  Val. PPL:   1.694\n",
      "Epoch: 281 | Time: 0m 2s\n",
      "\tTrain Loss: 0.197 | Train PPL:   1.218\n",
      "\t Val. Loss: 0.540 |  Val. PPL:   1.716\n",
      "Epoch: 282 | Time: 0m 3s\n",
      "\tTrain Loss: 0.195 | Train PPL:   1.215\n",
      "\t Val. Loss: 0.501 |  Val. PPL:   1.651\n",
      "Epoch: 283 | Time: 0m 2s\n",
      "\tTrain Loss: 0.201 | Train PPL:   1.223\n",
      "\t Val. Loss: 0.503 |  Val. PPL:   1.654\n",
      "Epoch: 284 | Time: 0m 3s\n",
      "\tTrain Loss: 0.198 | Train PPL:   1.219\n",
      "\t Val. Loss: 0.496 |  Val. PPL:   1.642\n",
      "Epoch: 285 | Time: 0m 3s\n",
      "\tTrain Loss: 0.195 | Train PPL:   1.215\n",
      "\t Val. Loss: 0.507 |  Val. PPL:   1.661\n",
      "Epoch: 286 | Time: 0m 2s\n",
      "\tTrain Loss: 0.194 | Train PPL:   1.214\n",
      "\t Val. Loss: 0.534 |  Val. PPL:   1.706\n",
      "Epoch: 287 | Time: 0m 2s\n",
      "\tTrain Loss: 0.193 | Train PPL:   1.213\n",
      "\t Val. Loss: 0.491 |  Val. PPL:   1.633\n",
      "Epoch: 288 | Time: 0m 3s\n",
      "\tTrain Loss: 0.194 | Train PPL:   1.214\n",
      "\t Val. Loss: 0.505 |  Val. PPL:   1.657\n",
      "Epoch: 289 | Time: 0m 2s\n",
      "\tTrain Loss: 0.185 | Train PPL:   1.203\n",
      "\t Val. Loss: 0.500 |  Val. PPL:   1.648\n",
      "Epoch: 290 | Time: 0m 3s\n",
      "\tTrain Loss: 0.186 | Train PPL:   1.205\n",
      "\t Val. Loss: 0.471 |  Val. PPL:   1.601\n",
      "Epoch: 291 | Time: 0m 3s\n",
      "\tTrain Loss: 0.190 | Train PPL:   1.209\n",
      "\t Val. Loss: 0.480 |  Val. PPL:   1.616\n",
      "Epoch: 292 | Time: 0m 2s\n",
      "\tTrain Loss: 0.190 | Train PPL:   1.209\n",
      "\t Val. Loss: 0.499 |  Val. PPL:   1.647\n",
      "Epoch: 293 | Time: 0m 2s\n",
      "\tTrain Loss: 0.188 | Train PPL:   1.207\n",
      "\t Val. Loss: 0.534 |  Val. PPL:   1.706\n",
      "Epoch: 294 | Time: 0m 3s\n",
      "\tTrain Loss: 0.186 | Train PPL:   1.204\n",
      "\t Val. Loss: 0.548 |  Val. PPL:   1.730\n",
      "Epoch: 295 | Time: 0m 3s\n",
      "\tTrain Loss: 0.187 | Train PPL:   1.206\n",
      "\t Val. Loss: 0.532 |  Val. PPL:   1.702\n",
      "Epoch: 296 | Time: 0m 3s\n",
      "\tTrain Loss: 0.186 | Train PPL:   1.204\n",
      "\t Val. Loss: 0.557 |  Val. PPL:   1.745\n",
      "Epoch: 297 | Time: 0m 3s\n",
      "\tTrain Loss: 0.174 | Train PPL:   1.190\n",
      "\t Val. Loss: 0.529 |  Val. PPL:   1.697\n",
      "Epoch: 298 | Time: 0m 2s\n",
      "\tTrain Loss: 0.183 | Train PPL:   1.201\n",
      "\t Val. Loss: 0.530 |  Val. PPL:   1.699\n",
      "Epoch: 299 | Time: 0m 3s\n",
      "\tTrain Loss: 0.180 | Train PPL:   1.197\n",
      "\t Val. Loss: 0.504 |  Val. PPL:   1.655\n",
      "Epoch: 300 | Time: 0m 2s\n",
      "\tTrain Loss: 0.177 | Train PPL:   1.194\n",
      "\t Val. Loss: 0.519 |  Val. PPL:   1.681\n"
     ]
    }
   ],
   "source": [
    "n_epochs = 300\n",
    "clip = 0.1\n",
    "\n",
    "model_path = os.path.join(os.getcwd(), \"model.h5\")\n",
    "\n",
    "best_valid_loss = float('inf')\n",
    "\n",
    "for epoch in range(n_epochs):\n",
    "    \n",
    "    start_time = time.time()\n",
    "    \n",
    "    train_loss = train(model, train_iter, optimizer, loss_slot, loss_intent, clip)\n",
    "    valid_loss = evaluate(model, val_iter, loss_slot, loss_intent)\n",
    "    \n",
    "    end_time = time.time()\n",
    "    \n",
    "    epoch_mins, epoch_secs = epoch_time(start_time, end_time)\n",
    "    \n",
    "    if valid_loss < best_valid_loss:\n",
    "        best_valid_loss = valid_loss\n",
    "        torch.save(model.state_dict(), model_path)\n",
    "        \n",
    "    print(f'Epoch: {epoch+1:02} | Time: {epoch_mins}m {epoch_secs}s')\n",
    "    print(f'\\tTrain Loss: {train_loss:.3f} | Train PPL: {math.exp(train_loss):7.3f}')\n",
    "    print(f'\\t Val. Loss: {valid_loss:.3f} |  Val. PPL: {math.exp(valid_loss):7.3f}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
